Solving Limited-Memory BFGS Systems with Generalized Diagonal Updates

In this paper, we investigate a formula to solve systems of the form (Bk + D)x = y, where Bk comes from a limited-memory BFGS quasi-Newton method and D is a diagonal matrix with diagonal entries di,i ≥ σ for some σ > 0. These types of systems arise naturally in large-scale optimization. We show that provided a simple condition holds on B0 and σ, the system (Bk + D)x = y can be solved via a recursion formula that requies only vector inner products. This formula has complexity Mn, where M is the number of L-BFGS updates and n M is the dimension of x. We do not assume anything about the distribution of the values of the diagonal elements in D, and our approach is particularly for robust non-clustered values, which proves problematic for the conjugate gradient method.